Task Based User Testing 
of Collections on Archive. ora 


Findings from December, 2015 



Overview 
of Findings 


The good 

• Users were able to quickly distinguish between collection and item tiles. 

• All users were able to locate the download area on the details page. 

• Users liked the large images in the media header of details pages. 

The not so good 

• Users had issues locating specific filters or filter types on the collections 
and search pages: 5 out of 7 testers were unable to locate language 
filter (one of the two successful testers was an employee of Archive.org) 

• Many users had a hard time locating the filter/sort bar on the collections 
and search pages, especially the sort arrows. Additionally, the dates 
dropdown was unexpected and distracting for many users. 

• Collection and sub collection hierarchy was confusing for users. Users 
did not expect to see (sub)collections interspersed with items on 
collections and search pages. 

• Poor image quality on collection tiles was distracting for users and 
made the site feel less trustworthy. 



High-level themes 

• Users want to feel more in control of their ability to sort and filter 
collections and search results pages. Although the mechanics for much 
of what users wanted to do was available, it was hard for testers to 
locate, understand and utilize these features. 

• The underlying structure and collection hierarchy of the Archive was 
confusing for many users. This affected their understanding of what 
could be uploaded to the Archive, what they felt the would be able to 
find on the Archive, methods of search and browsing as well as the 
trustworthiness of items uploaded on the Archive. 

• Our current tools for browsing (collections and media type collections) 
do not meet the expectations of users. Collection provenance held 
varying importance for users and media types were hard for some users 
to understand (eg. what about images in books?). 


Overview 
of Findings 



Five users and two internal staff members were 
interviewed in December of 201 5. Testers were asked to 
perform nine tasks on the search and collections pages in 
a 15-45 minute session (in-person and remote). 

The goals of the session were to evaluate: 

• How the user utilizes the navigation bar, search or browsing on the 
homepage to find a specific collection? 

• Can the user successfully use the sorting and filtering features on the 
collections page? 

• Can the user successfully navigate to a collection's 'About Page'? 

• Can the user successfully download an item? 

• Can the user successfully locate a media-focused collection? 


About 
the Test 



Jo 

Novice User 
UX Designer 


Ted 

Novice User 
Engineer 


Li z 

First Time User 
UX Designer + Librarian 


Kelsey 

First Time User 
Librarian 


Mikael 

Expert User 
IA Employee 


David 

Expert User 
Museum Employee 


Chloe 

Expert User 
IA Employee 


Dive-in to 
the Sessions 


*Names have been changed 



Task 

Analysis 


Homepage 

• Almost all of the users located the "American Libraries" collection by 
scrolling through the list of Featured Collections on the homepage. It 
should be noted though, that most users felt that the visibility of this list 
was limited and that they would not have found it unless asked to 
explore (scroll through) the page as a part of the task. 

Collections Page 

• Users self reported a difficulty rating from 0 (impossible) to 10 (easy) for 
the tasks of 'determining how items in a collection are currently 
ordered' and 'sorting the collection by the date they were added to 
Archive.org with most recently uploaded items first.' The average for 
these tasks across users was 6 (not easy), with one user failing to 
complete the tasks. Visibility of the sort bar, ambiguous language and a 
lack of visual affordances were the primary obstacles for users. 

• Some users were confused by the mixing of items and sub-collections 
on collections pages. This confusion lead them to be unsure of what 
type of page type they were on when viewing collection pages. 



Task 

Analysis 

Continued 


Filtering 

• Users had issues locating specific filters or filter types on the collections 
and search pages: 5 out of 7 testers were unable to locate language 
filter (one of the two successful testers was an employee of archive.org) . 
The length of the fully exposed filter list was the main cause of trouble 
for users who had to scroll past large sets of topics to find additional 
filter types (eg. Subject vs Language). 

• The default ordering of filters (frequency of use) was ambiguous to 
many users (who initially assumed alphabetical ordering) and the 
affordances related resorting were not clearly understood. 

Item Detail Page 

• Media size is one of the top concerns of users on the Details page. 
Users generally liked large images, and wanted books to be 'readable' 
in the media header. Some users did expect to see basic metadata 
(title, date, creator) on top of or before the media. 

• Users generally had little to no problem to no problem locating the 
downloads area, but some were concerned about usage rights. 



Task 

Analysis 

Continued 


Browsing by Media Type 

• The software icon was confusing to some users, who thought it might 
be understood as 'save' by some visitors. All users relied on the 'hover 
text' associated with the iconography to identify each media type. 

• The 'top hat' on the navigation bar opening above the central 
navigation was so unsettling for two of the testers and caused them to 
need to abandon their current task. Both users had assumed that 
clicking on the media type icon would trigger a dropdown. 

• Labeling and collection hierarchy in the media type 'top hat' was 
confusing for 3 of the 4 users who utilized the navigation bar to locate a 
media type collection. Some users wondered if the listed collections 
were the only collections containing that media type on the Archive and 
the large avatars were appealing but ambiguous to users. The 
difference between 'Top' and 'Featured' was also troublesome, as users 
weren't sure of how these lists were curated. 



Key Insights 
and Findings 


Filtering 

Users had difficulty way-finding through the list of filters and filter types on 
Collection and Search pages. The length of the filter list, ambiguous 
affordances regarding sorting of filters and the design of current 'count' 
labeling created difficulty for all users and contributed to 5 out of 7 users 
being unable to fulfill the task of filtering content by a specific language. 


The truncation of facets under filter types 
would allow for more (or all) filter types (eg. 
Topics, Languages, etc) to be viewable 
simultaneously. Facets could be limited to 
five exposed facets at a time, with a 
paginated modal opening to expose the 
additional facets when 'More' is clicked. 
Aligning the frequency 'counts' per each 
facet increases both the visibility of the counts and also makes it easier to 
see how these numbers relate to each other (eg. ordered by amount). 


Recommendation 


Topics & Subjects 

Cactai 

15 

Yoga 

6 

Cats 

3 

Dance 

3 

Astrology 

3 


Suggested changes to filter feature 




Key Insights 
and Findings 
Continued 


Sorting 

Issues with general visibility of the sort bar as well as visual and textua 
affordances made it hard for users to comprehend how results were 
ordered and to know what parameters were available to them. 

Recommendation 
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Suggested changes to the sort bar 


The design above emphasizes the sort options that are currently applied by 
bolding the type as well as a color change. The sort 'direction' is sticky to 
the sort applied and the date option now has a visible affordance denoting 
additional menu items. Finally the view icons are textually labeled. 




Key Insights 
and Findings 
Continued 


Collection and Sub Collection Hierarchy 

The intermixing of collections and items on collection pages was 
disorienting for some users 


Recommendation 


20 SUB COLLECTIONS 


AJberta Homestead University of Alberta Cats glorious cats 

Records Libraries - The C I H M . . . 




Cat CATalog 



Meow! 



4.238 items 

1 

1 .230 items 

1 

734 items 

1 

712 items 

1 

340 items 


3,932 ITEMS 



Media Type 

O texts 1 5 

ll movies 6 

J audio 3 


Topics & Subji 

3CtS 


Cactai 


15 

f~1 Yoga 


0 

l~~l Cats 


3 

1 Dance 


3 

1 Astrology 


3 


Language 

1 english 

35 

DstG 



1885 - 2011 


Suggested changes to the collection page 


In this design sub collections are separated from item results on the 
Collection page. Additionally collection tile height is standardized as some 
users thought that larger collection tiles were a sign of a larger collection. 










Key Insights 
and Findings 
Continued 


Navigation Bar + Iconography 

The 'Top Hat' (clicking on a item opens above the navigation bar) was 
disorienting for some users, who expected the navigation bar to have a 
dropdown. The links and labeling inside of the Top Hat' created hierarchy 
and way-finding issues for a few testers, who were unsure of what the 
organizational principles represented regarding content type and quality 
(the difference between Top' and 'Featured' was especially unclear). No 
users utilized the navigation search bar during testing, some even returned 
to the homepage to start a new query. Legibility of icons may also be a 
problem as users relied primarily on 'helper text' to decode icons. 

Recommendation 

Further research on the functionality of the navigation bar, as well as 
testing of any potential prototype designs. 

Higher visibility (expansion) of the search bar, a 'sticky' navigation bar, 
reorganization of submenus and increased text should be explored. 



Additional Testing 

Testing with different user types will help us to understand what aspects of 
the Ul are difficult for users at different stages in their use and 
understanding of Archive.org. 

Some exploratory testing with brand new users (who had never heard of 
Archive.org) was performed in January, however additional testing with 
new users and testing with power users, partner institutions and other user 
types would be helpful. 

A/B Testing of Suggested Designs 

Designs built off of these user tests could be tested with other users, or 
internally and then tested in the wild against the current website using an 
A/B testing framework. A/B testing would allow for quantitative data to be 
collected on the potential and current designs. 


Next Steps 
for Collections 



Questions, comments, or concerns? 

Please don't hesitate to reach out to me on Slack, 
in person or by email at carolyn@archive.org 



